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Content  Analysis  in  Systems  Engineering  Acquisition 

Activities 


Karen  Holness — is  an  Assistant  Professor  in  the  Department  of  Systems  Engineering  at  the  Naval 
Postgraduate  School  (NPS)  in  Monterey,  CA.  She  holds  a  BS,  MS,  and  PhD  in  industrial  engineering 
from  the  University  at  Buffalo.  Prior  to  NPS,  she  worked  as  a  Navy  Civilian  in  the  acquisition 
workforce  for  eight  and  a  half  years  in  various  industrial  engineering,  systems  engineering,  and 
human  systems  integration  roles.  She  also  previously  worked  for  three  years  as  an  Industrial 
Engineer  at  Corning,  Incorporated  in  Corning,  NY.  [kholness@nps.edu] 

Abstract 

This  paper  examines  the  role  of  content  analysis  in  systems  engineering  technical  evaluation 
processes.  Content  analysis  is  a  qualitative  data  analysis  methodology  used  to  discover 
consistencies,  inconsistencies,  themes,  and  trends  within  datasets.  This  methodology  is 
particularly  useful  when  evaluating  Contract  Data  Requirements  List  documents,  as  well  as 
deficiency  reports  from  test  and  evaluation  activities;  examples  of  such  analyses  are 
provided.  Factors  that  can  impact  a  systems  engineer’s  ability  to  effectively  and  efficiently 
use  this  analysis  method  are  also  discussed.  Research  into  the  development  of  valid, 
relevant,  and  repeatable  analysis  criteria  promises  to  define  (1)  how  content  analysis  can  be 
used  consistently  across  different  system  baselines  and  (2)  how  content  analysis  results 
generated  during  the  “Production  and  Deployment”  and  the  “Operations  and  Support” 
acquisition  lifecycle  phases  can  be  used  to  shape  requirements  definitions  for  system 
upgrade  or  modification  contracts  and  new  baseline  contracts.  Finally,  content  analysis 
training  and  skill  development  for  systems  engineers  in  the  acquisition  workforce  is 
discussed. 

Introduction 

During  the  different  phases  of  a  system’s  lifecycle,  systems  engineers  evaluate  a  lot 
of  data  from  a  variety  of  sources.  A  key  part  of  analyzing  this  data  is  discovering  patterns 
and  using  those  patterns  to  support  additional  analyses.  As  stated  in  the  International 
Council  on  Systems  Engineering  (INCOSE)  Handbook, 

Systems  thinking  captures  and  exploits  what  is  common  in  a  set  of  problems 
and  corresponding  solutions  in  the  form  of  patterns  of  various  types  ... 
Systems  engineers  use  the  general  information  provided  by  patterns  to 
understand  a  specific  system  problem  and  to  develop  a  specific  system 
solution.  (INCOSE,  2015) 

A  variety  of  quantitative  and  qualitative  methods  exist  to  (1)  capture  or  generate  data 
needed  for  a  particular  analysis,  (2)  reduce  the  data,  (3)  evaluate  the  data  to  find  patterns, 
and  (4)  draw  conclusions  about  the  System  Of  Interest  (SOI).  This  paper  focuses  on  content 
analysis,  a  qualitative  method  that  is  well  suited  for  datasets  that  contain  primarily  text- 
based  data. 

What  Is  Content  Analysis? 

Patton  (2015)  describes  content  analysis  as  “any  qualitative  data  reduction  and 
sense-making  efforts  that  takes  a  volume  of  qualitative  material  and  attempts  to  identify  core 
consistencies  and  meanings.  ...  The  core  meanings  found  through  content  analysis  are 
patterns  and  themes.”  As  defined  by  Krippendorff  (2004),  content  analysis  is  used  to  make 
“replicable  and  valid  inferences  from  texts  (or  other  meaningful  matter)  to  the  contexts  of 
their  use”  and  is  most  successful  when  evaluating  attributions,  social  relationships,  public 
behaviors  and  institutional  realities.  The  basic  steps  to  conducting  a  content  analysis  are 
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summarized  below.  See  either  Krippendorff  (2004)  or  Patton  (2015)  for  detailed  descriptions 
of  each  of  these  steps. 

1 .  Decide  what  data  sources  to  use  for  the  analysis.  These  best  fit  the  research 
questions  (unitizing). 

2.  Identify  a  representative  data  subset  to  analyze  (sampling). 

3.  Transform  the  raw  data  into  analyzable  data;  evaluate  and  interpret 
characteristics  within  and  between  data  elements  by  assigning  elements  to 
categories  based  on  an  observed  pattern  or  theme.  This  can  include  inter¬ 
rater  agreement  studies  for  the  categories  (recording/coding). 

4.  Evaluate  and  interpret  the  categorized  data,  looking  for  additional  patterns, 
themes,  correlations  (e.g.,  sub-categories)  and  outliers  (reducing  data). 

5.  Infer  the  meaning  of  the  categories.  Test  and  validate  the  inferences  with 
respect  to  the  research  questions  (inferring). 

6.  Summarize  and  communicate  the  analysis  findings  (narrating). 

Content  Analysis  in  Systems  Engineering  Activities 

Within  a  systems  engineering  context,  both  “attributes”  of  system  components  and 
“institutional  realities”  with  respect  to  operational  and  maintenance  concepts  for  a  given  SOI 
are  identified  and  evaluated  during  a  system’s  design  lifecycle.  Therefore,  someone  taking 
the  time  to  gather  existing  text-based  documents  from  either  electronic  or  paper  sources 
and  look  for  patterns  and  themes  is  already  done  within  systems  engineering  practice  to 
varying  degrees. 

One  example  is  the  case  where  various  stakeholders  and/or  representative  users  are 
interviewed  to  capture  their  inputs  on  what  the  SOI  needs  to  do  and  what  should  be 
reflected  in  the  corresponding  system  requirements  and  technical  performance  measures. 
The  answers  to  the  interview  questions  have  to  be  evaluated  and  summarized  in  some 
fashion.  Another  example  is  performing  trade  studies,  when  various  industry  information 
sources  are  reviewed  to  understand  the  latest  systems  available  on  the  market  and  current 
technology  trends  that  may  apply  to  the  SOI.  Reviewing  different  documents  or  websites, 
the  systems  engineer  looks  for  very  specific  features  and  compares  and  contrasts  them  in 
some  fashion.  The  INCOSE  (2015)  Systems  Engineering  Handbook  describes,  in  detail, 
each  of  the  standard  technical  processes  and  the  various  activities  that  take  place  within 
each  process.  Table  1  provides  a  sample  of  systems  engineering  activities  described  in  the 
handbook  that  most  likely  involve  the  review  of  qualitative  data  from  text-based  sources. 
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Table  1.  Sample  of  Systems  Engineering  Technical  Process  Activities  for 

Content  Analysis 

(INCOSE,  2015) 


Process 

Activity 

Business  or 

Mission  Analysis 

•  Analyzing  gaps  across  the  problem  or  opportunity  trade  space 

Stakeholder 

Needs  and 

Requirements 

Definition 

•  Eliciting  and  prioritizing  stakeholder  needs 

•  Identifying  solution  constraints  resulting  from  agreements  or  interfaces  with 
other  systems 

System 

Requirements 

Definition 

•  Ensuring  the  system  requirements  adequately  reflect  the  stakeholder 
requirements 

•  Negotiating  modifications  to  the  reguirements  to  resolve  any  issues  identified 

Architecture 

Definition 

Process 

•  Analyzing  "relevant  market,  industry,  stakeholder,  organizational,  business, 
operations,  mission,  legal,  and  other  information'  to  guide  architecture 
development 

Design  Definition 

•  Identifying  types  of  design  characteristics  for  each  system  element 

Systems  Analysis 

•  Comparing  results  from  several  types  of  models 

Implementation 

•  Eliciting  constraints  related  to  implementation  from  “stakeholders,  developers, 
and  teammates' 

•  Analyzing  and  resolving  any  anomalies  that  occurred  during  implementation 

Integration, 
Verification  and 
Validation 

•  Analyzing  and  resolving  any  anomalies  that  occurred  during  integration, 
verification  and  validation 

Operations 

•  Identifying  additional  operations  constraints  and  providing  feedback  to  the 
system  design  team 

•  Analyzing  and  resolving  any  anomalies  that  occur  during  operations 

Maintenance 

•  Identifying  additional  maintenance  constraints  and  providing  feedback  to  the 
system  design  team 

•  Analyzing  and  resolving  any  anomalies  that  occur  during  maintenance  activities 

•  Identifying  trends  in  maintenance  and  logistics  actions 

•  Eliciting  customer  feedback  on  maintenance  and  logistics  support  process 
satisfaction 

Of  particular  interest  to  this  paper  are  the  technical  processes  that  take  place  after 
System  Analysis.  The  Implementation,  Integration,  Verification  and  Validation  processes 
typically  span  the  “Engineering  and  Manufacturing  Development”  acquisition  lifecycle  phase 
As  described  in  Table  1,  all  of  these  processes  share  one  activity  in  common:  analyzing  and 
resolving  any  anomalies  that  occur  during  each  process’s  execution.  During  this  phase,  the 
product  baseline  for  the  SOI  is  reviewed  and  approved  during  the  key  Systems  Engineering 
Technical  Reviews  (SETRs)  that  are  required  prior  to  the  start  of  the  “Production  and 
Deployment”  acquisition  lifecycle  phase:  the  Critical  Design  Review  (CDR),  Test  Readiness 
Review  (TRR),  and  the  Production  Readiness  Review/Functional  Configuration  Audit 
(PRR/FCA). 

In  preparation  for  each  of  these  SETR  events,  the  contractor  systems  engineers 
evaluate  the  SOI’s  design  and  document  its  status  in  the  required  Contract  Data 
Requirements  List  (CDRL)  documents.  Examples  of  these  documents  are  Design 
Description  documents,  Product  Drawings  and  Associated  Lists  (PDALs),  and  Deficiency  or 
Discrepancy  Reports  (DRs).  These  documents  are  then  reviewed  and  interpreted  by  the 
Government  Systems  Engineers,  Logisticians,  and  Test  &  Evaluation  (T&E)  Engineers  for 
accuracy  and  validity  and  in  order  to  assess  the  SOI’s  adequacy  and  readiness.  Any 
anomalies  would  be  discussed  with  the  contractors  to  either  resolve  or  come  up  with  a 
mitigation  strategy,  preferably  before  the  SETR  event. 


v\  ,-|il  :;l  ^  // 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-59 


While  this  may  sound  simple  and  straightforward,  it  can  be  a  daunting  task,  even  for 
systems  with  relatively  few  components  and  for  documents  housed  in  a  configuration 
management  software  like  DOORS®.  Ensuring  consistency  within  and  across  Design 
Description  documents  requires  the  system  engineer  to  cross-reference  the  content  of  each 
document,  looking  for  specific  similarities  and  differences.  This  is  important  because  each 
document  focuses  on  detailed  aspects  of  the  same  SOI.  Similarly,  the  PDALs  may  contain 
hundreds  of  component  and  subsystem  drawings,  including  those  for  the  Commercial  Off 
the  Shelf  (COTS)  components.  The  contractor  systems  engineers  have  to  review  each 
component  drawing,  ensuring  that  the  content  makes  sense  and  correlates  with  the  other 
drawings  that  each  one  references.  The  interfaces  depicted  in  these  drawings  must  also 
match  the  same  interfaces  described  in  the  design  description  documents.  This  matching 
task  can  reveal  configuration  errors  that  could  impact  component  production  in  the  next 
acquisition  phase.  For  the  DRs,  it  is  the  responsibility  of  the  systems  and  T&E  engineers  to 
review  these  reports,  evaluate  them  for  patterns  and  themes,  and  interpret  what  those 
patterns  and  themes  reveal  about  the  performance  of  the  software  and  hardware.  The 
Government  Systems  Engineers,  in  an  acquisition  oversight  role,  independently  repeat  the 
same  process  for  each  one  of  these  documents. 

The  systems  engineer,  as  the  Subject  Matter  Expert  (SME),  will  be  held  accountable 
for  the  hardware  and  software’s  performance  by  the  Program  Manager.  Looking  for  trends, 
correlations,  and  consistencies/inconsistencies  helps  the  systems  engineer  evaluate  the 
feasibility  of  the  technical  baseline  and  qualify  the  reliability  and  quality  of  the  data  used  in 
the  evaluation.  For  the  Government  Engineers,  this  kind  of  analysis  also  helps  gauge  the 
quality  of  the  contractor’s  technical  performance. 

The  Operations  and  Maintenance  (O&M)  processes  described  in  Table  1  correspond 
to  the  “Production  and  Deployment”  and  the  “Operations  and  Support”  acquisition  lifecycle 
phases.  Like  the  previous  technical  processes  discussed  above,  the  O&M  processes  also 
analyze  and  resolve  any  anomalies  that  occur.  Tracking  system  performance  measures  and 
periodically  correlating  that  data  to  deficiency  log  data  or  maintenance  action  reports  can 
reveal  additional  factors  that  are  impacting  system  performance.  Similar  to  the 
Implementation  process,  it  is  important  that  any  additional  constraints  observed  by 
users/operators,  maintainers,  other  engineers  or  stakeholders  within  the  O&M  processes  are 
captured,  documented,  and  evaluated.  Once  fed  back  to  the  system  designers,  this 
information  can  then  be  used  to  shape  requirements  definition  for  system  upgrade  or 
modification  contracts  and  new  baseline  contracts. 

It  is  important  to  note  that  the  systems  engineers  doing  data  analysis  in  the  O&M 
processes  may  be  different  people  than  the  ones  who  worked  on  the  contract  in  previous 
phases  of  the  acquisition  lifecycle.  Instead  of  working  for  the  program  office  or  the  prime 
contractor,  these  systems  engineers  may  work  for  the  installation  site  and  are  responsible 
for  capturing  and  analyzing  system  performance.  Evaluating  and  packaging  these  data  and 
data  analysis  results  can  be  a  different  task  if  it  is  being  done  to  support  local  management 
or  will  be  provided  to  an  outside  organization  for  system  design  purposes. 

Research  on  the  Use  of  Content  Analysis  in  Systems  Engineering  Activities 

Systems  engineers  seem  to  be  performing  some  level  of  content  analysis.  But,  in 
which  technical  activities?  How  “well”  is  it  being  done?  How  valid  are  the  results?  Valuable 
insight  can  be  gained  by  researching  the  actual  use  of  content  analysis  in  the  technical 
processes  previously  discussed  and  what  software  tools  are  used  and  can  be  used  to 
facilitate  the  process. 
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For  example,  Fortune  and  Valerdi  (2013)  developed  a  framework  for  determining 
how  to  reuse  previously  created  engineering  products  for  a  new  development  effort.  As  part 
of  the  evaluation  phase  in  this  framework,  the  first  step  is  to  analyze  both  internally  and 
externally  developed  products  that  are  available,  like  requirements  documents  or  modeling 
tools,  then  determine  whether  or  not  they  apply  to  the  SOI.  Because  this  seems  to  involve  a 
comparison  of  a  previous  system  to  the  new  SOI,  an  investigation  into  the  effectiveness  of 
using  content  analysis  categories  may  prove  to  be  useful.  Since  the  next  step  in  this 
framework  is  to  estimate  the  costs  and  anticipated  benefits  from  reusing  the  engineering 
products,  having  supporting  evidence  generated  from  a  thorough  content  analysis  may  help 
to  justify  the  investment. 

It  is  easily  hypothesized  that  the  successful  use  of  content  analysis  as  a  research 
methodology  within  a  design  environment  or  an  operational  setting  would  be  impacted  by 
factors  such  as 

•  Time  required  and  resource  availability  to  spend  on  the  analysis 

•  Familiarity/Expertise  with  content  analysis  methods 

•  Familiarity/Expertise  with  the  technical  subject  matter  and  data  content 

•  Data  access,  particularly  when  data  are  spread  across  multiple  print  and 
electronic  sources 

•  Data  quality/quantity 

•  Individual  personality — having  the  ability  and  patience  to  search  for  and 
identify  patterns  in  datasets  of  various  sizes 

It  would  be  worth  researching  the  impact  that  content  analysis  would  have  on  the 
system  engineer’s  workload.  Such  a  study  could  provide  supporting  evidence  for  hiring  a 
dedicated  data  analyst  on  acquisition  projects  to  perform  various  technical  content  analyses. 
Investigating  the  degree  to  which  the  other  factors  listed  above  actually  impact  content 
analyses  can  help  identify  constraints  and  possible  mitigations  to  support  the  use  of  this 
methodology  in  different  acquisition  phases. 

Another  possible  research  path  is  the  development  of  valid,  relevant  and  repeatable 
analysis  criteria  that  can  be  used  across  different  system  baselines.  Granted,  every  system 
is  unique.  However,  research  to  develop  either  (1)  appropriate  contexts  and  levels  of  depth 
for  content  analysis  efforts  within  different  acquisition  phases  or  (2)  generalizable 
categorizes  for  system  attributes  would  help  lay  a  foundation  for  integrating  this 
methodology  into  the  systems  engineering  toolkit.  Having  a  common  analysis  tool  that  is 
easy  to  use  would  support  the  feedback  of  observed  system  performance  trends  from  the 
operational  and  maintenance  community  to  the  design  community,  which  would  be  used  to 
develop  requirements  for  system  upgrade  or  modification  contracts  and  new  baseline 
contracts. 

Finally,  the  implications  of  content  analysis  on  training  and  skill  development  for 
systems  engineers  in  the  acquisition  workforce  should  be  investigated.  Frank  (2006) 
evaluated  interview  and  survey  data  (using  content  analysis  as  part  of  his  data  analysis 
methodology)  to  characterize  the  cognitive  characteristics  and  abilities  of  engineers  with  a 
high  Capacity  for  Engineering  Systems  Thinking  (CEST).  While  the  ability  to  identify  patterns 
and  themes  was  not  specifically  identified  in  this  study,  the  characteristic  of  understanding 
analogies  and  parallels  between  systems  and  the  ability  to  conduct  trade  studies  were 
identified.  As  an  analysis  methodology  that  specifically  targets  these  abilities,  it  would  be 
interesting  to  evaluate  use  of  content  analysis  on  the  development  or  enhancement  of  these 
abilities.  It  would  also  be  worth  developing  guidelines  to  use  content  analysis  specifically  in 
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baseline  comparison  analyses,  providing  training  on  its  use,  then  determining  any  impact  to 
the  perceived  validity  of  the  analysis  results.  Additional  studies  that  test  instructions  on  how 
to  identify  and  validate  data  sources,  gather  data  from  these  sources,  and  use  commonly 
available  software  tools  like  Microsoft  Excel  would  further  demonstrate  the  feasibility  of 
using  this  methodology. 
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Overview 


•What  is  Content  Analysis? 

•Sample  of  Systems  Engineering 
Technical  Process  Activities  for 
Content  Analysis 

•  Factors  that  can  impactthe  use 
of  content  analysis 

•  Future  Research  on  the  use  of 
content  analysis 


What  is  Content  Analysis? 

Krippendorff  (2004)  and  Patton  (2015) 

1.  Decide  what  data  sources  to  use  for  the  analysis 
These  best  fit  the  research  questions  (unitizing). 

2.  Identify  a  representative  data  subset  to  analyze 
(sampling). 

%  Transform  raw  data;  Evaluate  and  interpret 

characteristics  within  and  between  data  elements 
(recording/coding). 

4.  Evaluate  and  interpretthe  categorized  data  (reducing 
data). 

5.  Inferthe  meaning  of  the  categories.  Test  and  validate 
the  inferences  with  respectto  the  research  questions 
(inferring). 

6.  Summarize  and  communicate  the  ana  lysis  findings 
(narrating). 


Sample  of  Systems  Engineering 
technical  pro  cess  activities  for 
content  ana  lysis  (INC  OSE,  2005) 


Process 


System  Requirements  Definition 


Architecture  Definition  Process 


Systems  Analysis 


w 


Activity 


Ensuring  the  system  requirements 
adequately  reflect  the  stakeholder 
requirements 


Negotiating  modifications  to  the 
requirements  to  resolve  any  issues 
identified 


Analyzing  “relevant  market,  industry, 
stakeholder,  organizational,  business, 
operations,  mission,  legal,  and  other 
information"  to  guide  architecture 
development 

Comparing  results  from  several  types 
of  models 


Sample  of  Systems  Engineering 
technical  pro  cess  activities  for 
content  ana  lysis  (INC  OSE,  2005) 


Analyzing  and 

resolving 

anomalies 


Eliciting  or 
identifying 
constraints 
and/ortrends 


Implementation 

Integration, 

Verification 

^Validation 

Factors  that  can  impactthe 
use  of  content  analysis 

•  Time  required  and  resource  availability  to 
spend  on  the  analysis 

•  Familiarity/ Expertise  with  content  analysis 
methods 

•  Familiarity/ Expertise  with  the  technical 
subject  matte  rand  data  content 

•  Data  access,  particularly  when  data  are 
spread  across  multiple  printand  electronic 
sources 

»  Data  quality/ quantity 

»•  Individual  personality  -  having  the  ability 
and  patience  to  search  forand  identify 
patterns  in  data  sets  of  various  sizes 


Future  Research  on  the  use 
of  content  analysis 

»■  Investigate: 

The  degree  to  which  the  previous  factors  actually 
impact  content  analyses  use 

The  actual  use  of  content  ana  lysis  in  the  technical 
processes  previously  discussed,  and  what  software  tools 
are  used  and  can  be  used  to  facilitate  the  process 

The  impact  of  content  ana  lysis  on  the  system 
engineer's  workload 

•  The  development  of 

■►appropriate  contexts  and  levels  of  depth  for  content 
ana  lysis  efforts  within  different  acquisition  phases 

•  genera lizable  categorizesforsystem  attributes 

The  implicationsof  content  analysison  training  and  skill 
development  for  systems  engineers  in  the  acquisition 
workforce 


Questions? 


